Papers with unimodal model
Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts (2022.coling-1)
Copied to clipboard
| Challenge: | Recent multimodal information extraction approaches overestimate the significance of images. |
| Approach: | They propose a general data splitting strategy to divide social media posts into two sets to achieve better performance under information extraction models of the corresponding modalities. |
| Outcome: | The proposed method outperforms existing models on two different multimodal information extraction tasks. |
MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Vision and language models exploit unrobust indicators in individual modalities instead of focusing on relevant information in each modality. |
| Approach: | They propose a performance-agnostic multimodality score based on Shapley values that quantifies in which proportions a multimodal model uses individual modalities. |
| Outcome: | The proposed model can quantify in which proportions a multimodal model uses individual modalities for different tasks and datasets. |
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)
Copied to clipboard
| Challenge: | Current language models have been criticised for learning language from text alone without connection between words and their meaning. |
| Approach: | They propose to train models on more sources than text to provide the lacking connection between words and their meanings. |
| Outcome: | The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation. |